The first electron microscope was invented by Max Knoll and Ernst Ruska at the Berlin Technische Hochschule in 1930s.^[30] Marton then reported one of the earliest application on biological samples and noticed that the high-energy electron beams destroyed the native structures.^[1] For many years, the TEM images had been interpreted in 2D. The birth of 3D electron microscopy was attributed to Aaron Klug and his colleague in 1968.^[31] The key mathematical theory that makes the 3D reconstruction possible is the Central Section Theorem, which states that the Fourier transform of a 2D projection is identical to the central slice of the Fourier transform of the 3D object in the same orientation. This implies the possibility to reconstruct a 3D structure from sufficient views of 2D images that can fill up the Fourier coefficients. The goal can be obtained by tilting the stage or using dispersed particles with randomly distributed orientation.

Intrinsic symmetry was a very important factor for 3D reconstruction in early days. Actually, almost all early structural studies of biological samples were based on the 2D crystals or highly symmetric samples, such as filament or icosahedral viruses.^[32] These targets were easier at the beginning for several reasons, such as easy orientation determination and reduced radiation damage by averaging identical copies. Before the birth of cryo-EM, the biological samples had been analyzed by staining approaches with heavy atoms.^[33–35] Nevertheless, there were clear limitations using such approaches because the structural details were destroyed during staining.

In 1975, Henderson and Unwin determined the structure of bacteriorhodopsin at 7 Å resolution in its native state as 2D crystals by using glucose to replace the solution.^[36] This allowed clear visualization of transmembrane helices and demonstrated the potential power of EM to reveal high resolution structures of biological molecules. But further improvement seemed almost impossible for biological molecules at room temperature at that time. Scientists were therefore groping for new methods that could potentially reduce the radiation damage. Freezing the biological samples using liquid nitrogen looked very promising to preserve high resolution structural information.^[37] But there were severe contamination problems using liquid nitrogen and the aqueous solution tended to form ice crystals during freezing.

The real birth of cryo-electron microscopy was attributed to the work by Dubochet and his colleagues in 1980s.^[4,38–40] Dubochet developed a most successful method to vitrify biological solution. In his simple but revolutionary method, biological solution is rapidly plunged into liquid ethane or propane and frozen as a thin film on a holey EM grid. The native structure of biological molecules can be successfully preserved in the vitreous ice and the radiation damage is largely reduced. This method has been widely used in modern researches. No doubt, it was the key progress that finally gave the real birth to cryo-EM technique. Fifteen years later after the first publication of bacteriorhodopsin structure,^[36] Henderson et al. successfully determined the 3.5 Å resolution structure and built the first atomic model of this membrane protein by cryo-EM.^[41]

The anticipation of the cryo-EM method for studying general structures was gradually arising in the meantime. Joachim Frank is the pioneer who generalized cryo-EM to study isolated proteins without crystals. Frank’s method is exactly the one we are widely using today, the so-called ‘single particles analysis (SPA)’. In contrast to Henderson’s previous study on bacteriorhodopsin using 2D crystals, Frank aimed for the structure determination using dispersed ‘particles’ in solution. In his early career, he found it was possible to align two EM images to high accuracy by cross-correlation.^[42] By aligning and averaging the structurally identical particles, signal-to-noise ratio (SNR) could be greatly enhanced which allowed structural analysis in more details.^[43,44] In 1981, Frank was able to build up the earliest software package called ‘SPIDER’^[45] for SPA. These early studies had created the basis and built up the main frameworks of cryo-EM.

In spite of the possibility that EM could be applied to study 3D structures in general case, i.e. the samples in solution, single particle analysis was not a straightforward method at the beginning. Most early high resolution structures were determined using 2D crystals in 1990s.^[41,46–48] It was also pointed out that cryo-EM was very likely to become a general method for near-atomic structural studies of non-crystalline proteins^[41,49] at that time. However, several limiting factors, such as radiation damage, specimen movement, and charging etc. still remained very challenging before cryo-EM could finally reach its theoretical limit.^[49]

In additional to the critical limiting factors that largely affected the image quality,^[49,50] computational methods to obtain the initial model and determine the particle orientations used to be very difficult for single particle cryo-EM.

Frank’s early study in 1970s demonstrated the possibility to study non-crystalline proteins.^[42,44] However, study of isolated particles required more sophisticated computation. The representation of each EM image as a multidimensional vector by van Heel et al. created a number of possible methods afterwards, including multivariate statistical analysis (MSA) for classification.^[51] At that point, orientation determination in general cases still remained a challenging problem. A key progress was attributed to Harauz and Ottensmeyer, who originated the projection matching algorithm for orientation determination and refinement against an available 3D model.^[52] This algorithm has been the most popular method and the basis for most modern programs. However, a reliable initial model is still required to successfully match the calculated projections of the 3D map with the observed projections, i.e., the cryo-EM images. Based on some earlier common line approach on icosahedral reconstruction by Crowther et al.,^[32] a method for ab initio orientation determination on asymmetric objects was developed by van Heel et al., known as the ‘angular reconstitution’.^[53] At the same time, Radermacher et al. developed another well-known method, the ‘random conical tilt’ (RCT).^[54,55]

Once the concept of 3D reconstruction from random projections became clear, many new ideas began to blossom. In the late 1990s, several more programs were developed for general application of single particle cryo-EM, such as IMAGIC,^[56] XMIPP,^[57] EMAN,^[58] FREALIGN,^[59] and Bsoft.^[60] Many other stand-alone programs were also developed to deal with many technical problems in different parts.

The most important recent progress in cryo-EM field was probably the successful application of direct electron detectors (DED).^[6,26,29,61] Before that, film and charge-coupled device (CCD)^[62] were used for electron detection, which usually resulted in sub-nanometer resolution in well-off cases.^[14,63,64] Two main factors had hampered further development of these detectors for high resolution structure determination by cryo-EM. One was the low detective quantum efficiency (DQE) and the other was the incapability for recording movie frames. Compared to CCD which has a DEQ of ∼ 7–10% at half Nyquist, the film was usually better at high frequency, with a DQE of 30—35% at half Nyquist.^[65] This was probably the key reason why most high resolution cryo-EM structures^[66,67] were obtained on film before DED, with only a few exceptions on icosahedral viruses.^[68–71] The new generation of DED, based on complementary metal–oxide–semiconductor (CMOS) technology, has much better DQE and faster recording speed.^[6]

The conceptual innovation of movie recording provided an incisive solution for several critically important limiting factors in cryo-EM, including the beam induced movement.^[29,49,50] Due to the fast recording rate, individual electrons can be ‘counted’ in theory, given the probability of overlapped electrons approaches zero.^[61,72] The other great benefit from locating the centers of individual electrons is that physical pixels can be divided into smaller ‘logic pixels’. This operation is usually called ‘super-resolution’ that can further enhance the DQE of modern direct detectors.^[61,72]

The year 2013 was marked by a milestone work from Cheng’s group, who determined the first cryo-EM structure of the membrane protein TRPV1 at 3.4 Å resolution by using direct detector.^[25] This breakthrough inspired many scientists all over the world, not only in cryo-EM field, but also in X-ray crystallography and many other fields. The whole field has kept on expanding since then. Both methodology development and biological applications are exploding now.

The image contrast is one of the most important limiting factors for structural determination by cryo-EM. It has been well known that in-focus micrographs lack good contrast.^[73] A certain amount of defocus is required to enhance the contrast of EM images. Vitrified biological samples in cryo-EM usually require even higher defocuses, typically several microns, to gain sufficient contrast. This in turn makes the CTF determination and correction more challenging for high resolution information restoration. Longer exposure with higher doses can also enhance the contrast. But it then leads to severe radiation damage and beam-induced movement. Energy filter is another option for higher contrast by getting rid of the inelastic scattering during TEM imaging. However, its potential is limited and does not essentially break out of the technical restrictions from the conventional TEM imaging. Phase plate is an exceptional concept that is possible to achieve tremendous contrast enhancement.

Based on the assumption of weak phase object approximation (WPOA) in traditional TEM imaging, the CTF is a sine-like function in reciprocal space. As a result, the low frequency information is reduced to almost zero, thus greatly bringing down the image contrast. The idea of phase plate is to convert the sine-like function to cosine-like so that low frequency signal can be well preserved. In other words, the aim is to find a proper approach that can be used to impose a 90° phase shift of the electron waves in the back focal plane. Zernike phase plate (ZPP) was firstly designed by Danev and Nagayama^[74] to achieve this goal in cryo-EM application. The design could be briefly described as a thin carbon film with a hole in the center. In spite of the enhanced contrast in several good examples,^[75,76] the disadvantage clearly emerged in the following years, such as contamination, short lifetime, fringe artefacts, and challenging alignment.

Voltage phase plate (VPP) is a recently-developed new version which consists of a thin (∼ 10 nm) continuous amorphous carbon film constantly heated to ∼200 °C.^[77] The phase shift is typically created ‘on the fly’ (by accumulation of exposures).^[77] This new design successfully overcame several drawbacks that had limited the application of previous ZPP. In the past few years, it has been used to successfully determine several structures of very small proteins to high resolution,^[78,79] including the 64 kDa (1 Da = 1 u = 1.66054 × 10⁻²⁷ kg) myoglobin at 3.2-Å resolution^[80] and a class B GPCR–G-protein complex.^[81] By using VPP and Cs-corrector as a combination, Fan et al. discovered the equivalence between under-focus and over-focus.^[82]

The VPP technique is now being widely used for high resolution study on many targets that could have never been imagined to be determined by single particle cryo-EM method. If we regard the application of direct detector in cryo-EM the critical progress for ‘resolution revolution’, VPP is probably nothing less than the revolution once more.

In addition to hardware development, substantial progress on data processing and software development also plays equally important roles in cryo-EM. On major challenge in single particle cryo-EM is the orientation. Classical cross-correlation approach, or least square method using normalized images, had several key problems, such as severe overfitting and model-bias. The failure is attributed to the extremely low SNR of cryo-EM images and unreliable references. Maximum likelihood (ML) approach in cryo-EM image analysis was first introduced by Sigworth^[83] to improve the accuracy of alignment on noisy images. Instead of assigning a single value for each parameter of individual particles, the basic concept of ML is to assign probabilities for all possible values of the unknown parameters. The objective is to estimate the ‘most likely model’ that best fits the assumption and observations (i.e., experimental data). The likelihood is usually calculated under the assumption that noise and unknown parameters obey certain distributions, e.g. Gaussian distribution for noise and (X, Y) origins, even distribution for in-plane rotation etc. The target is to maximize the likelihood function, typically in the form of logarithm, which can be achieved in an iterative way, called the ‘expectation-maximization’ approach. The expectation step assigns the probabilities for undetermined parameters in the presence of a currently available model; the maximization step estimates a new model based on previous expectation of parameter probability distribution.

The ML method was then developed by Scheres et al. and extended to tackle a more challenging problem, the conformational heterogeneity.^[84] It was originally implemented in the software package XIMPP. Later, a Bayesian approach^[27] was implemented in the program RELION^[28] for cryo-EM structure determination by optimization of a single function. Because of its user-friendliness and satisfactory results in most cases, this program is now prevailingly popular in the cryo-EM field. One of the limitations in old version of RELION was the cost of computation as a computer cluster was required for general application. The latest version, RELION2,^[85] takes the advantage of the Graphics Processing Unit (GPU) to accelerate many of its kernel functions. This significantly relieves the computational burden for many cryo-EM labs. Recently, a much faster software package, cryoSPARC,^[86] was released to further accelerate single particle data processing. In most cases, a single GPU workstation is competent to process an entire dataset. The great speed was achieved by taking advantage of GPU and faster algorithms, the stochastic gradient descent (SGD) and branch-and-bound maximum likelihood optimization.

In addition to better alignment and classification algorithms, the application of DED probably created more opportunities in cryo-EM field, which allows the movie processing and sophisticated weighting schemes. Specimen movement, including both mechanical drift and beam-induce motion, is one of the most important limiting factor for high resolution structural studies.^[50] Because of much faster recording speed, movies can now be recorded at a dose below 1 e/Ų per frame. Grigorieff’s lab first showed that tens of frames from the same movie stack can be accurately aligned to correct the movement of each individual particle, which dramatically restored high resolution structural information.^[29] A follow-up publication demonstrated that the Fourier coefficients of each frame can be precisely weighted according to the accumulated doses and frequencies.^[87] This ‘dose filtering’ approach can be regarded as pre-reconstruction weighting of particle frames. Another post-reconstruction approach is implemented in RELION, called ‘particle polishing’,^[88] in which the weights are typically calculated using the estimated B-factors from each frame reconstruction. Due to the very low electron doses per frame and high noise, motion correction on each particle alone proved to be very challenging and inaccurate. Li et al. reported a program called MotionCorr^[61] for frame level motion correction, based on least square method. This has been a very useful and successful tool in cryo-EM field, which generate nice estimation of the drift overall and is user-friendly. However, least square fitting usually does not guarantee smooth estimation of the drifts among all frames. Also, local drift correction and dose weighting were not available in previous version of MotionCorr.^[61] An improved variant of this program, the MotionCor2,^[89] was designed to deal with these problems and released recently. In most practical cases, the results were reported to be better, such as the representative work on TRPV1.^[25,90]

Apart from the important problem of unknown orientations, the 2D projections of a 3D object are severely modulated by the contrast transfer function (CTF) in TEM.^[91,92] In theory, if all the parameters, such as defocus, astigmatism and phase shift could be accurately recorded during imaging, the information could be restored via deconvolution of the CTF. However, this is not the case in cryo-EM for practical reason. Therefore, accurate estimation of the unknown CTF parameters is essential for high resolution reconstruction.

Since the beginning of 2014, we have been developing a GPU-accelerated CTF determination and correction program, called GCTF. The original purpose at the beginning was to overcome the severe orientation preference of dynactin complex^[93] by tilting the stage to high degree. Accurate CTF estimation on each particle is the key solution. After a careful investigation, we found that bending of the support layers, beam induced movement, particle position in ice, charging etc. could all lead to the CTF variation. Therefore, we decided to write our own program to do accurate CTF estimation on individual particles. The local CTF refinement strategy with a weighting scheme in both Fourier and real spaces generates accurate local CTF parameters and avoids overfitting of noise. In addition, the very fast speed of GCTF allows the users to perform real-time evaluation of micrographs during the data collection. Many people in the cryo-EM field have benefited from our GCTF program. Since 2014, it has already been widely used via personal communication. Several early applications^[93–95] of GCTF were even published before the official publication and release of GCTF.^[96]

We will keep developing GCTF and related programs in the future. One plan is to generalize the concept of equiphase averaging (EPA) approach described in early version of GCTF for CTF determination and refinement on phase plate data, tilted micrographs and movie frames etc. Another major plan is to extend GCTF for thick samples in both cryo-EM and cryo-ET.^[97] When the sample is considerably thick, we must take the defocus gradient and variation into consideration in order to get higher resolution. In the most simplified model, defocus gradient is considered as a linear function of Z-height. Based on this assumption, one can simply compensate the defocuses in different local region by adding an offset of Z-height. In practice, however, the modulation of structural information of thick biological samples is more complicated. Many other factors are also critically important, such as multiple scattering, uneven accumulation or fluctuation of the charges on samples and on support layers, variable local potentials of the vitrified biological samples (e.g., differences among solution, lipids and proteins etc.), interaction between electron beams etc. There is no simple and magic trick to solve this problem merely by adding an offset of Z-height to the estimated defocus.

What is more important, the projection is no longer a linear integration if the sample is too thick. For very thin samples under the assumption of strict WPOA, one can consider EM images in two different perspectives: i) linear projection first and then CTF modulation; ii) CTF modulation first and then linear projection. They are essentially identical operations because the phase shift of the exit wave is considered as instant effect. The order of the CTF modulation and linear projection can be swapped without any concerns. However, the two operations severely affect each other on a considerably thick 3D object. Back-projection (BP) algorithms or its variants, such as weighted back-projection (WBP), must be based on linear integration. These analytical algorithms are incompetent to decompose the severely CTF-modulated and non-linearly overlapped signals. Whenever a slice at a certain Z-height is CTF-corrected by adding a defocus offset to the averaged defocus estimation, signals from all other slices with different Z-heights are incorrectly deconvoluted by completely wrong CTF information. We proposed to tackle this problem together with algebraic reconstruction techniques (ART) several years ago, particularly the filtered interactive reconstruction technique (FIRT, first developed in 2008).^[98] Unlike WBP, the ART algorithm and its variants are perfectly suitable to address such CTF problem on thick 3D objects. We will discuss a bit more about these issues elsewhere in the future.

Of course, there are many other aspects of technical development that contributed to the reformation of modern cryo-EM. It will not be possible to discuss every nut and bolt of the cryo-EM technique in details in this length-limited review. In the following sections, we will try to provide several personal perspectives in cryo-EM.

It can be expected that methods on all parts of cryo-EM, i.e. sample preparation, data collection and data processing, will be further improved in the near future. More and more types of cryo-EM grids^[99–101] will probably be designed to increase the efficiency of sample adsorption, improve the particle distribution and reduce the beam induced movement etc. Equipment for fast sample freezing will be further improved to capture intermediate states for time-resolved structural studies at second or millisecond scale. Microscopes themselves can also be improved in multiple aspects, such as better mechanical stability, more coherent beams and faster imaging speed etc. Direct detectors will be further optimized with built-in counting algorithms and much improved DQE. In spite of the dramatic progress of VPP,^[79] we believe there is still ample room for optimization so that an almost perfect 90° phase shift can be imposed to the electron waves without additional background. If this goal were archived, the micrograph contrast of even more challenging targets, such as tomographic tilt series on sub-cellular context, can be further enhanced by many times. This will possibly allow more accurate alignment of individual particles inside the cell with very low electron doses, say 1 e/Ų–2 e/Ų.

In addition to the kernels of cryo-EM, the modern technology innovation in all other related fields will also make tremendous differences. Computers will be faster and cheaper to allow daily structure determination on individual workstations or even personal desktops. The GPU revolution will further reduce the computational cost and the latest cards (e.g. NVIDIA Titan V) can easily exceed the performance of an entire computer cluster 10–15 years ago. The classical image processing algorithms can still be improved, while more intriguing opportunities may emerge from other modern concepts like ‘deep learning’ and ‘compressive sensing’ etc. Together with these advances, highly automatic data collection, structure determination and analysis are already on the agenda.

What we can imagine is that conventional structure determination by cryo-EM will gradually become a routine, easy and straightforward task. Structures of protein complexes will probably be determined on-the-fly and even in batches. Systematic screening of chemical compounds for drug design will be expectable as well. All these intriguing prospects are arousing more and more excitements in this field. Many exciting, inspiring and unexpected results will be exploding in the near future, for instance, structures determined at super-high resolution, network contacts among a whole bunch of protein complexes, complete structural mechanisms of the entire pathway of a specific system, in situ structural studies in sub-cellular or even cellular context, dynamic structure determination on extremely flexible targets, so on and so on.

In classical cryo-EM technique, the reconstruction is always performed in 3D. With the fast development of cryo-EM, now is the time to seriously consider about determining dynamic structures of protein complexes at high resolution. Not only the structures in vitro will be suitable for dynamic analysis, but also the in situ systems will be deeply explored. That is the real power of cryo-EM. Single particle analysis and electron tomography will probably be both useful for different purposes. In the past few decades, they have been developed separately; in the following years, there will be great opportunities to combine multiple techniques together to answer really fundamental biological questions, probably in the cellular context. We would expect to see more breakthroughs in both the resolution and complexity of the biological targets. The really cool expectation is that we are now ready to visualize the molecules as dynamic machinery in their native states and even in the cell. As there are too many interesting possibilities to be discussed in details, we will have to focus on the dynamic structure analysis in the following parts of this review as a representative point for the future cryo-EM technique.

Before we discuss what can be done to achieve these goals, we need to clarify several concepts so as not to mislead the beginners in this field. First of all, what do we mean by dynamic structure determination after all? It should be clarified that dynamic structure by cryo-EM is not the same concept as ‘direct visualization’ of the active objects, just like we observe the living cells under light microscopy. Because all molecules in cryo-EM have been frozen into the vitreous ice, they are essentially immobile if the tiny thermal motion at atom level is ignored. When we mention the dynamic structural analysis or 4D reconstruction from the 2D projections by cryo-EM, we mostly mean computational approach to recover and sequence all conformational snapshots of a certain biological molecule or complex. These snapshots approximate a continuous and smooth conformational space given sufficient sampling. How could it be possible? The answer is actually very simple. Just think about a perfect dancer, who is repeating exactly the same dance again and again. You take photos randomly and capture a certain gesture on each acquisition. If you are allowed to take millions or even billions of pictures whenever you want to, it will be very likely to cover the whole choreography. Structural biologists are able to do the same thing on molecules by cryo-EM. On each micrograph, you can take pictures for tens or hundreds of individual proteins showing different conformations at different orientations. Given sufficient particle numbers, it becomes possible to recover the whole conformational space. With much better and faster equipment, such goal is no doubt expectable in the future.

We should also emphasize that the dynamic structural analysis by cryo-EM is also different concept from molecular dynamics (MD) simulation. The MD simulation is basically theoretical prediction of the behaviors of a molecule in solution, while dynamic structure determination by cryo-EM is completely experimental analysis. We do use computational approach to recover the dynamic structural information in cryo-EM, but there is nothing to be simulated. The computation in cryo-EM is only a tool to process the experimental data. The aim of cryo-EM is to obtain all possible conformations just from experimental data alone. Although it is possible to take advantage of MD simulation in the intermediate state of data analysis (e.g. to predict the preferred motion), no prior knowledge from theoretical prediction should be imposed in the final reconstruction. This is very important to make sure that the dynamic structural information is essentially experimental via computational approaches. We will discuss in more details about the possible approaches for practical implementation in the following section.

Successful analysis of a highly heterogeneous system to reveal the protein dynamics requires preservation of structural information as much as possible. This is an even more challenging objective than obtaining a static high resolution map for atomic model building by cryo-EM. There are many aspects of cryo-EM technique that can lead to the loss of structural information. These can be briefly classified into three major groups: sample preparation, imaging and computation. The information loss in each major group is also a combined effect of multiple sub-factors. We briefly summarize what factors can be improved or need to be considered.

(I) Sample preparation

This includes purification of the targets, the optimization of solution environment and the cryo-EM grid preparation. They are all very important and can make big differences depending on the specific application. Both compositional and conformational heterogeneity can be introduced during the protein purification step or by post-purification treatment, such as cross-linking, considerable times of concentration or dilution, freezing and thawing (for storage purpose), changes of salt or pH or temperature, detergent or other chemical compounds added to the solution and so on. The sample quality can also be severely affected during the freezing, including the types of grid used, treatment of the grid and conditions for freezing. These factors can significantly change the absorption and orientation distribution of the protein complexes. The air-water interface is probably the most disastrous region for the preservation of biological structures.^[2,102] The strong surface tension of the thin solution film introduced during the blotting can lead to severe distortion of protein complexes, sometimes completely destroying the targets. We need functional heterogeneity and conformation distribution to finally reveal the dynamic structures for understanding the mechanisms. But on the other hand, we must try all our best to avoid the artificial or non-physiological heterogeneity. Otherwise, all the rest work on the structural analysis will be merely in vain. Therefore, careful sample preparation is almost the basis for all the following study that guarantees the quality of cryo-EM structures.

In addition to the quality of structural information, factors that can affect the quantification of the conformation distribution of the targets should be taken into serious consideration as well. Almost all the aforementioned factors can make more or less differences. In the future, if we want to disclose the elaborated mechanisms of a certain complex at quantifiable precision, this is very important to make sure the final computational result is a true reflection of the nature itself rather than artificial or biased. So far, there are almost no literatures discussing such quantitative effects during experiments and its effects on the following computational analysis. Further investigation and efforts are needed to address these issues. Obviously, to obtain a dynamic structure with quantifiable precision probably requires much more data than what is needed for a static 3D structure.

In most cases, the biological sample solution can be regarded as an equilibrium system. All conformations are assumed to co-exist in the system and they can be captured in the same grid simply by normal freezing methods using equipment like Vitrobot etc. Computational approach is sufficient to reveal the conformational distribution from such a system given sufficient data. We do not need to resort to time-resolved techniques experimentally. However, if we want to make clear what is happening to the whole system during an irreversible reaction, i.e., the target is not in an equilibrium state, ‘time-resolved freezing methods’ must be required. This is still a very challenging topic at the current stage since most biochemical reactions are too fast to be controlled for structural studies. The methods like Spotiton^[103] and mixing-spraying^[104] are representative and inspiring approaches, but still not suitable to capture structures in action instantly. More elaborate methods by taking advantage of special materials or extreme conditions may be worth thinking about in the future.

(II) Cryo-EM imaging

Recent development in cryo-EM has manifested the power of direct electron detectors to record much improved micrographs. We will soon expect even better products to be launched into market. The DQE curve of future detectors is very likely to be near 1.0 at almost all frequencies except near-Nyquist. Is it possible to record perfect images using such perfect detectors? The answerer is probably very disappointing. It is definitely impossible using current cryo-EM technique. Even a perfect detector would not allow 100% recording of the structural information from biological samples. The key reason is that most information is actually lost before detection rather than during detection, if a modern direct detector is used for data collection. In order to clarify this issue, we define two consecutive processes during the entire TEM imaging, the ‘pre-detection’ and ‘detection’. The pre-detection represents the period of time from the moment electrons are emitted by the FEG to the point they have penetrated the biological samples, ready to be sensed by detectors. The detection refers to the process from the moment electrons hit the surface of the detector to the point they are sensed, recorded and quantized as digital signals. The DQE of direct detectors only connotes how much information we can preserve during the detection process. It has nothing to do with the loss of information during the pre-detection process.

The information loss during pre-detection is attributed to many factors, such as the irreversible physical changes of biological samples during imaging, random thermal motion of molecules, imperfect optical system of microscope, mechanical instability, image formation on the vulnerable biological targets with considerable sizes and all other factors that violate the analysable theories. The beam-induced movement, ionizing radiation, accumulating or fluctuating charges, etc. can all contribute to irreversible physical changes of the samples. Thermal motion of molecules is inevitable even if the cryo-EM grids are preserved under liquid nitrogen temperature. In addition to the overall movement of large patches induced by electron beams, thermal motion of individual water molecules can be severely enhanced by electron beams. Therefore, the ‘signals’ of water molecules are turned into pure noise because of beam-enhanced thermal motion. This eventually brings more troubles for later motion correction. The thickness of biological samples can be a problem under the assumption of weak-phase object approximation. The defocus gradient, dynamic scattering and effective contrast all become considerably serious problems on large samples (e.g. > 100 nm), particularly when the required resolution becomes higher. At a lower voltage (e.g. 200 kV instead of 300 kV), the effect is even stronger. Not only the structural information of the specimen itself is attenuated, but also the electron beams can be affected by the structural changes and charging effect during imaging, thus further reducing the image quality. These factors are nothing related to detectors. We believe there is still vast space to be improved during the pre-detection process.

Once scientists are able to make a clear mathematical description of the physical process during cryo-EM imaging, it might be very likely to recover as much information as possible by well-designed computational approaches afterwards. In principle, it would be possible to recover almost all structural information from the highly noisy, severely modulated and radiation-damaged cryo-EM images. If not, it is mainly because we have not established a perfect physical model with a perfect estimation of all required parameters. That is exactly our future endeavour. We hope in the future the major uncertainty left will be the quantum effect of biological molecules during TEM imaging. But that is probably far beyond the objective we can think about at current stage.

(III) Computation

We will discuss possible computational approaches separately.

In order to get continuously moving structures from cryo-EM data, we must make a clear picture on the intrinsic property of functional conformational changes of biological complexes. Two simplified mathematical models can be applied to deal with the heterogeneous datasets. In the classical model, the conformational changes are regarded as movement of multiple rigid bodies. In the other more generalized model, the conformational changes are interpreted as elastic deformation. Both can be useful in different cases. The first model is simpler for programing. The latter model seems a bit more complicated, but represents the generalized case. To be more precise, the latter actually incorporates the multibody model as one special case.

In the case of multiple bodies with discrete conformations or assemblies, conventional classification methods can be very useful. Although there are a variety of methods that already exist to deal with conformational heterogeneity, the concepts are all deeply rooted in 3D analysis. Most of the clustering methods are essentially designed to tackle the discrete conformations. The entire dataset is typically classified into several individual classes and then refinement is simply performed on each dataset in the same way as homogenous dataset. Examples include separation of the 8-fold and 9-fold symmetric chaperonins^[63] and dynactin complex with the p150 projection density.^[93] The popular program RELION has shown its great power to sort out many discrete conformations and is currently widely used in the cryo-EM field.

Local refinement of the orientations for each individual domain will also be very helpful. The goal can be archived by applying a mask on the interesting region. When the targeted local density is big enough, the calculated local projection and experimental 2D images of the entire molecule can be accurately aligned to improve the local resolution. However, when the interesting region becomes smaller and smaller, false peaks will emerge because of the extremely noisy image and the non-zero correlation with the projections from other uninteresting regions of the 3D map. This significantly affects the accuracy and correctness of the local orientation. The signal subtraction approach was proposed to partially address this problem.^[105,106] The approach is implicitly based on the assumption that the alignment of the major part of a flexible structure is accurate enough in the presence of small local flexibility. The cross-correlation (or any other types of similarity functions) can then be successfully maximized (or minimized) on the local region by avoiding its correlation with the signals of the major region. This approach has been successfully applied in both cases of local flexibility^[107,108] and symmetry-mismatch reconstruction.^[71,105] In practice, each individual subunit or domains or any other types of user defined local regions could be iteratively improved by subtraction and re-subtraction of different local regions to further improve the local alignment accuracy. Overfitting of local alignment is probably inevitable when an interesting region becomes too small. In such a case, local classification without alignment might generate better results, but still not fully guaranteed.

More realistically, no protein complex can be a perfect multibody system; they are intrinsically elastic and eternally dynamic. For such a reason, signal subtraction approach is not suitable in the general cases of continuous conformational changes as the implicit assumption of multibody does not hold any more. The subtraction approach becomes even challenging when the complexity of the system become huge and completely uncontrollable. The concept of manifold can be borrowed from mathematics to perfectly describe such conformational changes of protein complexes with some modifications. A manifold in mathematics can be regarded as the generalized concept of curves or curved surfaces and locally resembles Euclidean space. The concept of manifold itself is nothing new and can be traced back to the beginning of 19th century. The application of this basic mathematical concept was first introduced to natural science by physicists to study the curved space-time of the universe. The manifold methods in image processing was only proposed less than 20 years ago.^[109] In cryo-EM, it only began to draw the attention from structural biologists a few years ago.^[110] The concept can now be used to well describe the continuous motion of biological macromolecules.

It should be noted that even if the basic concept is similar, there are always different perspectives to understand the same natural phenomena. The manifold concept that describes protein motion in cryo-EM can be interpreted in several apparently distinct ways. In one point of view, a cryo-EM image with N pixels (typically 10⁶–10⁹) is mapped to a single point in an N-dimensional Euclidian space. Different points in this high-dimensional space represent different conformations. The continuous motion of a protein can then be regarded as a smooth manifold embedded in this N-dimensional Euclidian space. This perspective has both advantages and disadvantages. The concept is clear, simple and makes sense. Machine learning algorithms for manifold embedding in high-dimensional space is well developed in computer science and there are many available resources to implement it as well. However, one big issue is that some very useful information of cryo-EM images is discarded or reduced when a 2D image is completely discretized as a single point in N-dimensional space. For example, the local continuity and smoothness in the original 2D image are both lost. It therefore becomes very challenging to apply geometric constraints among certain regions of a density map in high-dimensional space. Dimensional reduction approaches have to be exploited to handle these problems. The most important disadvantage is that the manifold embedding approach in high-dimensional space lacks a good sense of ‘geometrical intuition’ for human beings to understand the natural motion of proteins.

We would rather suggest a different perspective, the probabilistic conformational space (PCS), which is also based on the manifold approach for cryo-EM structural analysis in the case of continuous motion. It is essentially a low-dimensional manifold assigned with a probability function for each point of time, plus some additional structures (to be discussed later). Multiple time axes (say 4–10) are necessary to describe more complicated motions. Typically, the time axes are continuously and smoothly curved. Such perspective does not necessarily require the embedding of manifold into a high-dimensional space, but tries to reveal the implicit properties of the manifold itself, such as local curvature and deformation parameters. We believe the best approach is to directly build a target function between the PCS and the observed cryo-EM images, instead of classification, sub-classification, sub-sub-classification, and so on.

If we regard the dynamic structure determination by cryo-EM as a unified target, then the aim is to assign each observed 2D image with a group of parameters (x, y, α, β, γ, t) by minimizing the following target function:

This point of view can also be interpreted as a generalized concept of multibody approach in cryo-EM data analysis, in the case of infinitely many rigid bodies, each of which is infinitely small. It is apparently not feasible to divide a map into too many rigid bodies. But the concept of PCS is competent to deal with such problem. All the rigid bodies along the time axes share 100% mutual information no matter how small they are. This extremely useful information can be implicitly estimated and fully exploited in the PCS. The concept of 3D reconstruction is then extended to the question of how to estimate the PCS. The classical concept of linear projections of a 3D density map is also generalized to be a non-linear integration in PCS. We regard this integration as non-linear because the projection paths are curved. The classification question will no longer exist, because discrete conformations are simply regarded as a special probability distribution, the Dirac delta function and its variants.

In practice, the reconstruction of the low-dimensional manifold and estimation of probability distribution might not be accurate due to the very low SNR in cryo-EM datasets. An optimized weighting scheme will be very helpful for better reconstruction. We believe there will be great opportunities for such optimizations based on the PCS concept. The following properties will be very useful if they are taken into consideration and treated properly.

I) Density conservation

This means that the integration of the density map for a specific molecule is constant for any conformation. The assumption might not be useful for the whole molecule as the cryo-EM images are always scaled and normalized to the same level in practical data processing. The real power is that any partial structure holds this property as well, which can be extremely useful to constrain the relative scales of local density maps among all individual regions. In theory, each region can be infinitely small. In practice, because of the difficulty in extracting useful information from very small region in the highly noisy cryo-EM maps, each region may require at least several residues.

II) Homeomorphism and homotopy equivalence

This property is based on the fundamental nature of proteins whose conformational changes are usually smooth and reversible (we do not discuss sub-atom or quantum effect at current stage). For any two conformations of exactly the same protein complex, there exists a continuous mapping that preserves all the topological properties and has an inverse mapping. And for each point in a protein density map, there also exists a smooth path (or curve) from any conformation to another. To be more concrete and specific, a protein can smoothly change its conformation from any state A to any other state B and vice versa (not necessary to be exactly the inverse path). All conformations of a protein are topologically identical, recyclable and reproducible in a smooth manner. This mathematical model also implies that once a protein conformation has ever existed in the past, it is likely to appear again after sometime, no matter how long it will take and how small the probability is.

III) Isometry

We define two types of isometric deformation. One is called ‘strong isometry’ to describe the fact that the lengths of all chemical bonds of proteins are relatively conserved during the dynamic conformation changes. Therefore, the distance between two atoms is a constant by tracing the backbone in any conformation. The chemical bond vibration at quantum level can be completely ignored at the resolution of 2 Å–3 Å by using current cryo-EM method. This kind of isometry can be concretely interpreted as ‘invariant sequence distance’, which is very different from the variable Euclidian distance between two atoms in 3D dimensional space during a conformational change. However, for a fully folded protein, the Euclidian distance between two atoms in a local region can also be roughly confined, though not as strict as the sequence distance. We call this property the ‘weak isometry’.

These aforementioned properties are mathematically very useful for analysing dynamic structures by cryo-EM. They can significantly simplify the computational approaches if properly exploited.

We have briefly discussed the factors that may affect the dynamic structure analysis by cryo-EM. It has been clearly demonstrated in both history and recent cryo-EM development that experiment, computation and theory are all critically important in practice. They will all contribute to dynamic structural analysis by cryo-EM. Experimentally, we need decent functional evidences or proofs from biochemistry, molecular biology, cell biology, genetics, and probably physiological analysis as well. For instance, each nucleotide binding state of dynein motor itself is very complicated and flexible;^[107] the intermediate conformational changes from one state to another are even more difficult to be captured. It would be more efficient to lock or constrain such flexibility to some extent by using biochemical approaches as initial investigation. It is also very useful to investigate the processive movement by light microscope. Although the resolution is not as good as cryo-EM, light microscope is almost free of radiation damage and can be used to visualize the living cell or tissue at nanometer resolution with a much larger field of view. This useful information is far beyond the reach of cryo-EM at the current stage. With the information being accumulated, it will then gradually become realistic to study the whole functional cycle. All types of experimental data can be converted to well-designed structural constraints in the following computational analysis. Computational information itself is contributed by two distinct resources, the cryo-EM reconstruction and MD simulation. These two can be well combined to generate better results. Cryo-EM maps can be used to constrain and regulate the MD simulation, while the latter can help cryo-EM predict the most likely motion and deformation involved in the continuous conformational changes. However, none of such constraints can be useful without a solid support from successful theories. Unlike mathematics and physics, the theoretical analysis in biological systems is so far still the weakest point, partially because of their huge complexity. Therefore, when we are being intoxicated by solving all kinds of fancy biological structures, more and more efforts are required to establish reliable and useful theories for daily experimental and computational analysis. A deep analytical understanding of the complicated biological imaging will be very helpful, particularly the extremely large samples. Ultimately, we will have to combine the tripartite information in order to obtain dynamic structures: experiments, computation and theories.